Introduction¶
The dataset contains 110,527 medical appointments of patients in various neighbourhoods in Brazil. It contains 14 observations that contain patient data. The last observation records whether or not, the patient attended the appointment.
Dataset Description¶
The observations are as follows:
- Patient Id : A unique patient identifier
- Appointment Id : A unique identifier for each appointment
- Gender: M or F to denote Male or Female
- Scheduled Day: Date some registered the appointment
- Appointment Day: Date patient is required to attend the appointment
- Age: Age of the patient
- Neighbourhood: Location of the appointment
- Scholarship: True or False to denote whether the patient is a recipient of the Bolsa welfare program
- Hipertension: True or False
- Diabetes: True or False
- Alcoholism: True or False
- Handicap: True or False
- SMS_received: Denotes whether patient a message through SMS prior to the appointment
- No-show: True or False
Question(s) for Analysis¶
- Which gender has the highest number of missed appointments?
- Are patients who are recipients of welfare likely to attend appointments?
- What day of the week has the most missed attendance?
- Does the number of waiting days affect attendance?
In [1]:
#Install missing packages using pip
#Restart Kernel afterwards
!pip install geopy;
!pip install --upgrade plotly;
!pip install --upgrade pyopenssl;
!pip install --upgrade certifi;
!pip install --upgrade pandas==1.1.5;
!pip install --upgrade tensorflow-tensorboard==0.1.1;
!pip --disable-pip-version-check install requests;
In [2]:
import warnings
warnings.simplefilter(action = 'ignore', category=FutureWarning) #removes warnings about future deprecations in libraries
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import geopy
import numpy as np
import plotly
%matplotlib inline
In [3]:
print("Pandas version " + pd.__version__) #requires minimum pandas 1.1.5
print("Geopy version " + geopy.__version__)
print("Plotly version " + plotly.__version__)
In [4]:
dataset = ('./Database_No_show_appointments/noshowappointments-kagglev2-may-2016.csv')
df = pd.read_csv(dataset)
df.head()
Out[4]:
In [5]:
df.info()
- ScheduledDay and AppointmentDay columns are objects and not in datetime format
Check for missing values¶
In [6]:
datatype = df.dtypes #checks the datatype of the column
sum_na = df.isna().sum() #the sum of any missing values in each column
na_ = df.isna().any() #checks for any missing values
info = pd.concat([na_,sum_na,datatype], axis=1, keys=['na_', 'sum_na','datatype'])
print(info)
- There are no missing values in the dataset
Check for duplicate records¶
In [7]:
duplicates = df.duplicated().sum()
print("There are {} duplicate records in the dataset ".format(duplicates))
Check the number of categories in each feature¶
In [8]:
for feature in df.columns:
print('{} has total {} categories \n'
.format(feature,len(df[feature].value_counts())))
- The handcap columns has 5 categories, instead of 2 categories (0,1) to represent True or False in the category.
- The number of Patient ID's indicates, some patients had more than one appointment.
Check the distribution of the features by plotting bar graphs¶
In [9]:
features_numeric = ['Age','Scholarship','Hipertension','Diabetes','Alcoholism','Handcap','SMS_received']
In [10]:
df[features_numeric].hist(figsize=(15,20));
Check the distribution of No-show column¶
In [11]:
df['No-show'].value_counts().plot(kind="bar",
xlabel='No show',
ylabel='Total Count',
title='No Show Distribution',figsize=(8,5));
- Majority of patients in the dataset attended their appointments.
Investigate the Age column¶
In [12]:
df['Age'].hist(figsize=(10,8), bins=15);
- Age distribution is skewed to the right,with majority of patients being under 60.
Check the unique value in the Age column¶
In [13]:
df['Age'].unique()
Out[13]:
Query the ages less than 0¶
In [14]:
df.query('Age <0')
Out[14]:
Query the ages equal to 0¶
In [15]:
df.query('(Age == 0) & (Alcoholism == 1)')
Out[15]:
- There is an an entry recorded in the age column as -1 and also 0. We are going to operate under the assumption that -1 represents an unborn child, while 0 represents a baby of less than 1 year.
- This assumption is supported by the fact that the record does not indicate any pre-existing conditions such as Diabetes and Alcoholism.
Investigate the Handcap column¶
Check the unique values in the column¶
In [16]:
df['Handcap'].unique()
Out[16]:
- There are 4 unique values in the column instead of 2.
Check the value counts of the 4 unique values¶
In [17]:
df['Handcap'].value_counts()
Out[17]:
Data Cleaning¶
Correct data in 'Handcap' column to remove values greater than 1, since 0 means handicap and 1 means handicap¶
The following function will convert any value above 1 to 1, and 0 will remain 0
In [18]:
def correct_handcap(x):
if x >= 1:
return 1
else:
if x == 0:
return 0
Apply the correct_handcap() function
In [19]:
df['Handcap'] = df['Handcap'].apply(correct_handcap)
df['Handcap'].value_counts()
Out[19]:
- The Handcap column now consists only of values, 0 and 1.
Change ScheduledDay, AppointmentDay datatype to datetime¶
In [20]:
df['ScheduledDay'] = pd.to_datetime(df['ScheduledDay'])
df['AppointmentDay'] = pd.to_datetime(df['AppointmentDay'])
- This columns are now in a datetime format
Create Weekday observation¶
In [21]:
df['App_weekday'] = df['AppointmentDay'].dt.dayofweek
- This observation will assist in the lambda operation that occurs in the next code cell determining whether the scheduled appointment occured on a weekday or weekend.
In [22]:
df['Part_of _week'] = df['App_weekday'].apply(lambda x:'weekend' if x >= 5 else 'weekday')
Derive names of the days of the week¶
In [23]:
df['App_dayname'] = df["AppointmentDay"].dt.day_name()
This has created a column containing names of the days of the week, Sunday to Monday.
Obtain the hour from the Scheduled Appointment¶
In [24]:
df['Schdl_hour'] = df['ScheduledDay'].dt.hour
A column containing the hour extracted from the time and date from the 'ScheduledDay' column has ben created.
Create observation that determines which part of the day, the scheduling appointment occured¶
The following function will categorise the different hours of the day according to the time
In [25]:
def get_duration (t):
if t >= 4 and t <= 9:
return 'Early morning'
elif t > 9 and t <= 12:
return ' Morning'
elif t > 12 and t <=13:
return 'Noon'
elif t > 13 and t <= 16:
return 'Afternoon'
elif t > 16 and t <=20:
return 'Evening'
elif t > 20 and t <= 24:
return 'Night'
elif t < 4 :
return 'Late Night'
In [26]:
df["Part_of_day"] = df['Schdl_hour'].apply(get_duration)
This function uses the hour value derived from the Scheduled Appointmnet time and contained in the Schdl_hour column to classify the time into;
Early Morning - (04:01 to 09:00)
Morning- (09:01 to 12:00)
Noon- (12:00 to 13:00)
Afternoon- (13:01 to 16:00)
Evening - (16:01 to 20:00)
Night - (20:01 to 00:00)
Late Night - (00:01 to 04:00)
Get the number of days between the Scheduling Day and Appointment Day¶
This is achieved by obtaining the difference in the number of days between the 'AppointmentDay' and 'ScheduledDay' and passing the values into the 'waiting_days' column.
In [27]:
df['waiting_days'] = (df['AppointmentDay'].dt.date) - (df['ScheduledDay'].dt.date)
df['waiting_days'] = df['waiting_days'].dt.days
Investigate 'waiting_days' column¶
In [28]:
df['waiting_days'].describe()
Out[28]:
- There exists a negative difference in days before the 1st quantile.
Query the data to see which records have a negative value in the 'waiting_days' column¶
In [29]:
df.query('waiting_days < 0')
Out[29]:
- There cannot exist a negative time difference since the patients, could not have attended their appointments, and then called in at a later day to book their previous appointment. This 5 records will be dropped from the dataset.
Query the records with a negative day difference to obtain one of the Patient Id's for further investigation.¶
In [30]:
df.query('waiting_days < 0').iloc[0,0]
Out[30]:
Using the specific Patient Id check for other appointments and their corresponding data.
In [31]:
df.query('PatientId == 7839272661752.0')
Out[31]:
- This specific Patient Id shows that the patient had other multiple appointments that occurred before and after the erroneous appointment recorded using AppointmentID : 5679978. All their other characteristics are similar.
Only the appointment with the negative 'waiting_day' difference shows a missed attendance. This is the same for all the records showing a negative difference.
This 5 records will be dropped from the dataset.
Use the same condition to drop the 5 records from teh dataset.
In [32]:
df = df[df.waiting_days >= 0]
Query the dataset to confirm the rows have been dropped.
In [33]:
negative_difference = len(df.query('waiting_days < 0'))
print("There are {} records that display a negative value in the 'waiting_days' column.".format(negative_difference))
Encode the 'No-show' column for better intuition.¶
In [34]:
df['Attendance'] = df['No-show'].apply(lambda x: 'Attended' if x == 'No' else 'Missed');
In [35]:
df['No-show'].value_counts()
Out[35]:
In [36]:
df['Attendance'].value_counts()
Out[36]:
In [37]:
df['No-show'] = df['No-show'].apply(lambda x: 0 if x == 'No' else 1);
Converted the 'No-show' column so as 1 denotes 'Yes' and 0 denotes 'No'
Get geolocation data for plotting Neighbourhoods¶
Obtain a list of all the neighbourhoods in the dataset
In [38]:
neighbourhood_list = df['Neighbourhood'].unique()
In [39]:
neighbourhood_list
Out[39]:
In [40]:
country = ', BRAZIL'
with_country = [neighbourhood + country for neighbourhood in neighbourhood_list]
Using the Geopy library obtain the latitude and longitude information of the neighbourhoods¶
In [41]:
!conda config --set ssl_verify False
In [42]:
%%time
from geopy.geocoders import Nominatim
locator = Nominatim(user_agent="kenneth_API") ##
geopy.geocoders.options.default_timeout = 1000
LAT = []
LONG = []
for x in with_country:
geolocation = locator.geocode(x, timeout=1000)
if geolocation is None: #checks whether the name matches to a location
LAT.append(None) #appends a nullvalue when location not foundd
LONG.append(None)
else:
latitude = geolocation.latitude #appends geodata to list
longitude = geolocation.longitude
LAT.append(latitude)
LONG.append(longitude);
Create a dataframe to store the location data¶
In [43]:
df_location_data = pd.DataFrame(list(zip(neighbourhood_list, LAT, LONG)), columns=['Neighbourhood', 'Latitude','Longitude'])
In [44]:
df_location_data.info()
Check for missing values¶
In [45]:
df_location_data['Latitude'].isnull()
Out[45]:
In [46]:
df_location_data.tail(10)
Out[46]:
- ILHAS OCEÂNICAS DE TRINDADE is not accesible through the Geopy library and therefore has null values
- According to Wikipedia, refers to an archipelago east of the coast of Brazil and forms part of the Brazilian state of Espírito Santo.
- The archipelago consists of five islands and several rocks and stacks; Trindade is the largest island, with an area of 10.1 square kilometres (3.9 square miles).
- For the purpose of plotting, the coordinates of the main island in the archipelago, 'Ilha de Trinidade' are going to be used inplace.
- Latitude : -20.524892
- Longitude : -29.324559
Drop the neighbourhood from the location data and replace it with "Ilha de Trinidade"¶
In [47]:
df_location_data = df_location_data[df_location_data.Neighbourhood != 'ILHAS OCEÂNICAS DE TRINDADE']
In [48]:
df_location_data = df_location_data.append(
{'Neighbourhood': ' ILHA DE TRINDADE','Latitude':-20.524892,'Longitude': -29.324559}, ignore_index=True)
In [49]:
df_location_data.tail(10)
Out[49]:
In [50]:
df.query('Neighbourhood == "ILHAS OCEÂNICAS DE TRINDADE"') #check records with that value
Out[50]:
In [51]:
df.loc[48754, "Neighbourhood"] = "ILHA DE TRINIDADE" #get indexes and use loc method to change
df.loc[48765, "Neighbourhood"] = "ILHA DE TRINIDADE"
In [52]:
df.loc[48754]
Out[52]:
In [53]:
df.query('Neighbourhood == "ILHA DE TRINIDADE"') #confirm the neighbourhood value is changed
Out[53]:
Create the observation 'Pre-existing condition'¶
This will display 0 if the patient lacks all four observations (Hipertension, Diabetes, Alcoholism and Handcap) and 1 if the have any of the four observations
In [54]:
df['Pre_existing_conditions'] = df['Hipertension'] + df['Diabetes']+ df['Alcoholism']+ df['Handcap']
In [55]:
df['Pre_existing_conditions'] = df['Pre_existing_conditions'].apply(lambda x:1 if x >= 1 else 0)
In [56]:
df['Pre_existing_conditions'].value_counts()
Out[56]:
In [57]:
df.head()
Out[57]:
Rearrange the columns to have 'No-show' and 'Attendance' at the end of the dataframe¶
In [58]:
cols = df.columns.tolist()
print(cols)
In [59]:
df = df[['PatientId', 'AppointmentID', 'Gender', 'ScheduledDay',
'AppointmentDay','App_weekday','Part_of _week',
'App_dayname', 'Schdl_hour', 'Part_of_day', 'waiting_days', 'Age', 'Neighbourhood', 'Scholarship',
'Hipertension', 'Diabetes', 'Alcoholism', 'Handcap','Pre_existing_conditions',
'SMS_received', 'No-show', 'Attendance']]
Plot the Neighbourhoods on a map¶
In [60]:
import plotly.express as px
fig = px.line_geo(lat=[0,15,20,35], lon=[5,10,25,30])
fig = px.scatter_mapbox(
df_location_data, # Our DataFrame
lat="Latitude",
lon="Longitude",
center={"lat":-4.047995, "lon":-40.864349},
width=1000, # Width of map
height=1000, # Height of map
hover_data=["Neighbourhood"])
fig.update_layout(mapbox_style="open-street-map")
fig.show()
- Most neighbourhoods are located in states that are in the east of Brazil and share the coastline
- The neighbourhoods are all in Brazil
- Ilha de Trinidade is an island in the Pacific Ocean, and only 2 appoitments from this area are recorded in the dataset.
In [61]:
df.head()
Out[61]:
Exploratory Data Analysis¶
Question 1: Which gender has the highest number of missed appointments?¶
Check the distribution of overall attendance
In [62]:
Attended = df.Attendance == 'Attended' #create masks
Missed = df.Attendance == 'Missed'
Get the number of unique Patient Id's and Appointment Id's
In [63]:
df['PatientId'].nunique() #unique patient Id's
Out[63]:
In [64]:
df['AppointmentID'].nunique() #unique number of Appointment Id's
Out[64]:
- There are 62299 unique Patient Id's that are linked to 110522 medical appointments.
In [65]:
df.Attendance.value_counts()
Out[65]:
In [66]:
plt.rc('font', size=15)
plt.rcParams['figure.figsize'] = [25, 7]
plt.bar(x=(df.Attendance.value_counts().index),height=df.Attendance.value_counts(), color='#c3d5e8')
plt.title('Appointment Attendance')
plt.box(False)
plt.grid(True)
plt.legend()
plt.show()
- Majority of appointments are attended.
In [67]:
attendance_by_percentage=(df.Attendance.value_counts()/df.Attendance.value_counts().sum())*100
attendance_by_percentage
Out[67]:
In [68]:
fig = plt.figure(figsize =(8, 8),tight_layout=False)
plt.pie(attendance_by_percentage, labels =('Attended','Missed'), colors=('turquoise','lime'),autopct='%1.1f%%');
plt.title('Attendance %')
plt.legend()
plt.show()
- Only 20.2% of all appointments are missed.
Get the number of Appointments by Gender
In [69]:
df.Gender.value_counts()
print("There are {} appointments attributed to female patients and {} appointments attributed to male patients"
.format(df.Gender.value_counts().values[0],
df.Gender.value_counts().values[1]))
Plot this distribution
In [70]:
fig = plt.figure(figsize =(10, 8),tight_layout=False)
plt.pie(df.Gender.value_counts(), labels =('Female','Male'), colors=('teal','purple'),autopct='%1.1f%%');
plt.title('Total Gender Distribution %')
plt.legend()
plt.show()
- The number of female patients is significantly higher at 65% while the percentage of men is 35%
In [71]:
df.Gender[Attended].value_counts() #attended appointments by gender
Out[71]:
In [72]:
print("Female patients attended {} while male patients attended {} of {} appointments"
.format(df.Gender[Attended].value_counts().values[0],
df.Gender[Attended].value_counts().values[1],
df.AppointmentID[Attended].value_counts().sum()))
In [73]:
fig = plt.figure(figsize =(10, 8),tight_layout=False)
plt.pie(df.Gender[Attended].value_counts(), labels =('Female','Male'), colors=('teal','purple'),autopct='%1.1f%%');
plt.title('Attended Appointments %')
plt.show()
- The ratio of attendance between female (64.9%) and male(35.1%) is similar to the overall gender distribution
- Females- comprise of 64.9% of attended appointments
- Males comprise of 35.1% of attended appointments
- There are more female patients.
- Female patients attended more appointments.
Check whether pre-existing conditions contribute to the higher number of female patients
In [74]:
with_pre_existing = df['Pre_existing_conditions'] == 1 #create mask
In [75]:
df.Age[with_pre_existing]
Out[75]:
In [76]:
by_male = df.Gender == 'M' #create mask
by_female = df.Gender == 'F'
In [77]:
df.Pre_existing_conditions.value_counts() #value counts for Pre-existing conditions
Out[77]:
In [78]:
print("There are {} patients with one or more pre-existing conditions while {} patients lack any pre-existing conditions"
.format(df.Pre_existing_conditions.value_counts().values[1],df.Pre_existing_conditions.value_counts().values[0]))
In [79]:
fig, ax = plt.subplots(1,2, figsize=(20,8))
ax[0].scatter(x=df.Age[by_female],
y=(df['Diabetes']+df['Hipertension']+df['Alcoholism']+df['Handcap'])[by_female],
alpha=0.8, color='teal', label='Female')
ax[0].set_xlabel("Age")
ax[0].set_ylabel("Number of Pre-existing conditions")
ax[0].legend()
ax[0].set_title("Distribution of Pre-existing Between Female and Male by Age")
ax[1].set_title("Distribution of Pre-existing Between Female and Male by Age")
ax[1].scatter(x=df.Age[by_male],
y=(df['Diabetes']+df['Hipertension']+df['Alcoholism']+df['Handcap'])[by_male],
alpha=0.5, color='magenta',label='Male')
ax[1].set_xlabel("Age")
ax[1].set_ylabel("Number of Pre-existing conditions")
ax[1].legend()
plt.show()
- Patients under 20 generally have only have one pre-existing condition.
- There is an increase in the number of pre-existing conditions as ge progresses.
- Patients who posses 3 pre-existing conditions, are found between the ages of 40 and 80.
Check to see if there is a relationship between Age and Gender
In [80]:
np.mean(df.Age[by_male]), np.mean(df.Age[by_female]) #mean ages by gender
Out[80]:
In [81]:
df.Age[by_male].describe() #descriptive statistics of age among men
Out[81]:
In [82]:
df.Age[by_female].describe() #descriptive statistics of age among women
Out[82]:
- Female patients have a mean age of 38.89
- Male patients have a mean age of 33.74
Plot distribution of age based on gender
In [83]:
plt.rc('font', size=15)
fig = plt.figure(figsize =(10, 8))
plt.hist(df.Age[by_male], bins=10, alpha=0.5, label='Male', color='purple',edgecolor='black')
plt.hist(df.Age[by_female], bins=10, alpha = 0.5, label='Female', color='teal',edgecolor='black')
plt.axvline(df.Age[by_male].mean(), color='y', linestyle='dashed', linewidth=5, label='Male mean')
plt.axvline(df.Age[by_female].mean(), color='r', linestyle='dashed', linewidth=5, label='Female mean')
plt.axvline(df.Age[by_male].median(), color='y', linestyle='dashdot', linewidth=1, label='Male median')
plt.axvline(df.Age[by_female].median(), color='r', linestyle='dashdot', linewidth=1, label='Female median')
x_ticks = [0,10,20,30,40,50,60,70,80,90,100,110,120]
plt.title('Age Distribution By Gender')
plt.legend()
plt.show();
In [84]:
from scipy.stats import skew
print("The skewness for Age of male patients is {}, and female patients is {}"
.format(skew(df.Age[by_male]),
skew(df.Age[by_female])))
- There are overall more female patients.
- There a more male than female children under the age of 10. This is the only age bracket where the number of males is higher than females
- Male age distribution is more skewed to the right than female age distribution. This signifies that the difference between the mean and median is greater in males than in females.
In [85]:
df.Gender[Missed].value_counts() #missed appointmnets by gender
Out[85]:
Plot bar graph showing distribution of missed appointments by gender
In [86]:
plt.figure(figsize=(25,10))
plt.bar(x=(df.Gender[Missed].value_counts().index),height=df.Gender[Missed].value_counts(), color='teal')
plt.title('Distribution of Missed Appointments By Gender', fontsize=15)
plt.xlabel='Gender'
plt.ylabel='Missed Appointments'
plt.box(False)
plt.grid(True)
plt.legend()
plt.show()
Plot pie chart showing distribution of missed appointments by gender
In [87]:
fig = plt.figure(figsize =(10, 8),tight_layout=False)
plt.pie(df.Gender[Missed].value_counts(), labels =('Female','Male'), colors=('teal','purple'),autopct='%1.1f%%');
plt.title('Missed Appointments %')
plt.show()
- The number of missed appointments by female patients: 14591
The numbr of missed appointments by male patients: 7723
Females comprise of 65.4% of missed appointments
- Males comprise of 34.6% of missed appointments
- Gender does not seem to affect attendance.
- Female patients missed more appointments.
Question 2: Are patients who are recipients of welfare likely to attend appointments?¶
In [88]:
df.Scholarship.value_counts()
Out[88]:
In [89]:
print("There are {} appointments of patients enrolled in the welfare program while the rest {} appointments are of patients not enrolled"
.format(df.Scholarship.value_counts().values[1],
df.Scholarship.value_counts().values[0]))
Plot bar graph to show welfare distribution among appointments
In [90]:
plt.figure(figsize=(10,7))
plt.bar(x=(df.Scholarship.value_counts().index[0]),height=df.Scholarship.value_counts().values[0], color='cyan', label='Non-Recipient')
plt.bar(x=(df.Scholarship.value_counts().index[1]),height=df.Scholarship.value_counts().values[1], color='plum', label='Recipient')
plt.title('No. of Appointments of Patients Receiving Welfare', fontsize=15)
plt.ylabel='Appointments'
ax = plt.gca()
ax.get_xaxis().set_visible(False)
plt.box(False)
plt.grid(True)
plt.legend()
plt.show()
Get the number of patients enrolled in the welfare program
In [91]:
with_welfare = df.Scholarship == 1 #receive welfare
no_welfare = df.Scholarship == 0 #not enrolled in welfare program
In [92]:
len(df.PatientId[with_welfare].unique()), len(df.PatientId[no_welfare].unique()) #apply condition to dataset
Out[92]:
In [93]:
patient_by_welfare=np.array([len(df.PatientId[with_welfare].unique()), len(df.PatientId[no_welfare].unique())], dtype='int64')
patient_by_welfare #contains distribution of welfare among patients
Out[93]:
Plot a pie chart to show the percentages of this distribution
In [94]:
fig = plt.figure(figsize =(10, 8),tight_layout=False)
plt.pie(patient_by_welfare, labels =('Recipient','Non-recipient'), colors=('indigo','pink'),autopct='%1.1f%%');
plt.title('Welfare Enrollment %')
plt.legend()
plt.show()
- There are 5788 patients enrolled in the welfare.
- There are 56511 patients not enrolled in the welfare program.
Check the relationship between enrolment in the welfare program and pre-existing conditions
In [95]:
df.Pre_existing_conditions[with_welfare].value_counts() # 0 for false, 1 for true
Out[95]:
In [96]:
df.Pre_existing_conditions[no_welfare].value_counts() # 0 for false, 1 for true
Out[96]:
In [97]:
fig, ax = plt.subplots(2,1)
fig = plt.figure(figsize =(20, 20),tight_layout=False)
ax[0].pie(df.Pre_existing_conditions[with_welfare].value_counts(),
labels =('No pre-existing conditions','Have Pre-existing conditions'),
colors=('darkorange','antiquewhite'),autopct='%1.1f%%')
ax[0].set_title('Pre-existing conditions among welfare recipients %', fontsize=15);
ax[1].pie(df.Pre_existing_conditions[no_welfare].value_counts(),
labels =('No pre-existing conditions','Have Pre-existing conditions'),
colors=('darkorange','antiquewhite'),
autopct='%1.1f%%');
ax[1].set_title('Pre-existing conditions among non-recipients %', fontsize=15);
- Patients enrolled in the welfare programs have less pre-existing conditions(Diabetes,Hipertension, Alcoholism, Handicap) when compared to patients who are not recipients of the program.
Investigate Appointment Attendance Among Welfare Recipients
In [98]:
df.Attendance[with_welfare].value_counts() #attendance of patients receiving welfare
Out[98]:
Plot a bar graph to show the distribution
In [99]:
plt.figure(figsize=(25,10))
plt.bar(x=(df.Attendance[with_welfare].value_counts().index),height=df.Attendance[with_welfare].value_counts(), color='teal')
plt.title('Attendance Patients Receiving Welfare', fontsize=20)
plt.xlabel='Gender'
plt.ylabel='Appointments'
plt.box(False)
plt.grid(True)
plt.legend()
plt.show()
In [100]:
fig = plt.figure(figsize =(10, 8),tight_layout=False)
plt.pie(df.Attendance[with_welfare].value_counts(), labels =('Attended','Missed'), colors=('teal','lightskyblue'),autopct='%1.1f%%');
plt.title('Attendance Patients Receiving Welfare %')
plt.legend()
plt.show()
In [101]:
fig = plt.figure(figsize =(10, 8),tight_layout=False)
plt.pie(df.Attendance[no_welfare].value_counts(), labels =('Attended','Missed'), colors=('teal','lightskyblue'),autopct='%1.1f%%');
plt.title('Attendance Patients Not Receiving Welfare %')
plt.legend()
plt.show()
- Patients not enrolled in welfare attended more appointments than those enrolled in the welfare programs.
Check the differences in attendance along age between men and women enrolled in welfare
In [102]:
by_age_attended_welfare = df.query('(Attendance == "Attended") & (Scholarship == 1)').groupby('Age')['PatientId'].count()
by_age_attended_welfare_fem = df.query('(Attendance == "Attended") & (Scholarship == 1) & (Gender == "F")').groupby('Age')['PatientId'].count()
by_age_attended_welfare_male = df.query('(Attendance == "Attended") & (Scholarship == 1) & (Gender == "M")').groupby('Age')['PatientId'].count()
In [103]:
by_age_attended_welfare.plot(label="Total")
by_age_attended_welfare_fem.plot(label="Female")
by_age_attended_welfare_male.plot(label="Male")
plt.title('Attendance count vs Age (Welfare Recipients)')
plt.legend()
plt.show();
-There are more male patients between 0 and 20, their number declines afterwards with a sharp rise and fall at around 60.
In [104]:
(df.query('(Scholarship == 1) & (Gender == "F")').groupby('PatientId')['AppointmentID'].count())
(df.query('(Scholarship == 1) & (Gender == "M")').groupby('PatientId')['AppointmentID'].count())
Out[104]:
In [ ]:
Question 3: What day of the week has the most missed attendance?¶
Check the distribution of scheduling of appointments
In [105]:
df['App_dayname'].value_counts() #distribution of appointments by weekday`
Out[105]:
In [106]:
df['App_dayname'].value_counts().plot(kind="barh",
ylabel='Days of week',
xlabel='No. of appointments',
title='Distribution of Appointments',
color='teal',
figsize=(8,6)
)
Out[106]:
- Saturday has the least number of appointments.
- There are no appointments on Sundays.
- The days ranked by total number of appointments in descending order are:
- Wednesday
- Tuesday
- Monday
- Friday
- Thursday
- Saturday
In [107]:
Attended = df.Attendance == 'Attended'
Missed = df.Attendance == 'Missed'
In [108]:
df.App_dayname[Attended].value_counts()
Out[108]:
In [109]:
df.App_dayname[Missed].value_counts()
Out[109]:
In [110]:
df.App_dayname[Attended].value_counts().sort_values()
Out[110]:
In [111]:
plt.rcParams['figure.figsize'] = [25, 15]
plt.bar(x=(df.App_dayname[Attended].value_counts().index),
height=df.App_dayname[Attended].value_counts(),
label='Attended', color='#24615D')
plt.bar(x=(df.App_dayname[Missed].value_counts().index),
height=df.App_dayname[Missed].value_counts(),
label='Missed', color='#00FFEF')
df['App_dayname'].value_counts()
plt.title('Appointment Attendance')
plt.box(False)
plt.grid(True)
plt.legend()
plt.show()
- Patients generally show up for appointments.
- Excluding Saturday, Thursday has the least number of missed appointments, followed by Friday. This may be due to their close proximity to the weekend.
- The trend is appointment attendance is the same in comparison to the total number of appointments for that day. This means Appointment Day does not affect attendance.
Check for any relationship between attendance and scheduling
In [112]:
df['Schdl_hour'].value_counts().plot.bar(figsize=(10,8),
color='g',
xlabel='Hours of the Day',
ylabel=('Scheduling of Appointments'),
title=('Scheduling of Appointments vs Time'));
Check Attendance when compared to registration/scheduling of appointments
In [113]:
by_percentange = (df['Part_of_day'][Attended].value_counts() / df['Part_of_day'].value_counts()) * 100 #attendance as a percentage by scheduling time
by_percentange
Out[113]:
In [114]:
by_percentange.plot(kind="barh",
title="Percentage Attendance");
- Noon has considerably less scheduling occuring in comparison to the hours, before and after. This is due to Noon being attributed a single, that occurs between(12:00 and 13:00).
In [115]:
df['Part_of_day'][Attended].value_counts()
Out[115]:
In [116]:
df['Part_of_day'][Attended].value_counts().plot.bar(figsize=(8,6),
color='g',
xlabel='Part of Day',
ylabel=('Attended Appointments'),
title=('Attended Appointments vs Scheduling:Part of day'));
In [117]:
df['Part_of_day'][Missed].value_counts() #missed appointments categorized by scheduling
Out[117]:
In [118]:
df['Part_of_day'][Attended].value_counts().plot.bar(figsize=(8,6),
color='g',
xlabel='Scheduling:Part of Day',
ylabel=('Missed Appointments'),
title=('Missed Appointments vs Scheduling'));
- Most appointments are registered in the morning.
- No scheduling occurs between 1 PM and 5 PM.
- No scheduling occurs past 10 PM.
- 7 am has the highest amount of scheduling occuring.
- People are least likely to schedule appointments during the night.
- There is less activity in terms of scheduling appointments as the day progresses.
Question 4: Does the number of waiting days affect attendance?¶
In [119]:
plt.figure(figsize=(8,6))
plt.hist((df.waiting_days[Attended]), label='Attended', alpha=0.8)
plt.hist((df.waiting_days[Missed]), label='Missed', alpha=0.7)
plt.grid(True)
plt.box(False)
plt.title('Waiting Days Before Appointment')
plt.xlabel="No. of days"
plt.ylabel="No. of Appointments"
plt.legend()
plt.show()
- The majority of waiting days are between 0 and 25 for attended appointments.
- The trend of waiting days is similar in both attended and missed appointments.
In [120]:
df.waiting_days[Attended].unique() #display range of missing days
Out[120]:
Check mean for waiting days categorized by attendance
In [121]:
np.mean(df.waiting_days),np.mean(df.waiting_days[Attended]),np.mean(df.waiting_days[Missed])
Out[121]:
In [122]:
fig = plt.figure(figsize =(10, 8),tight_layout=False)
y=np.array([9,16])
category = ('Attended','Missed')
explosion = [0.2, 0]
col = ('mediumspringgreen','plum')
plt.title('Mean of Waiting Days Before Appointment')
plt.pie(y, labels = category,explode=explosion, colors=col,shadow = True, startangle = 90)
plt.show()
In [123]:
print("The mean for all waiting days is {},\nthe mean for waiting days for attended appointments is {}\nand the mean for waiting days of missed appointments is {}"
.format((np.mean(df.waiting_days).round(0)),
np.mean(df.waiting_days[Attended]).round(0),
np.mean(df.waiting_days[Missed]).round(0)))
- The mean number of waiting days is 10 for all appointments.
- Attended appointments have a smaller mean of about 9 days
- Missed appointments have a mean number of about 16 days.
- Appointments with shorter number of waiting days have a higher attendance.
Conclusions¶
Female patients missed more appointments.
- The number of missed appointments by female patients: 14591
- The number of missed appointments by male patients: 7723
- Female patients comprise of 65.4% of missed appointments.
- Males comprise of 34.6% of missed appointments.
- Majority of appointments are attended, with only 20.2% of appointments being missed.
- There are 71837 appointments attributed to female patients and 38685 appointments attributed to male patients
- The number of female patients is significantly higher at 65% while the percentage of men is 35%.The ratio of attendance between female (64.9%) and male(35.1%) is similar to the overall gender distribution This lessens the impact of gender on attendance.
- There are 62299 unique patient id's that are linked to 110522 appointments.
- This means their exists repeat patients who had more than one appointment.
- There are 54154 uniue patient id's for attended appointments,and 17661 unique patient id's for missed appointments. This means that an overlap occurs where the same patients appear in both categories. It can be inferred that some of the patients who had more than one appointment,attended some appointments and missed others.
- There are 26410 patients with one or more pre-existing conditions(diseases) while 84112 patients lack any pre-existing conditions.
- Patients under 20 generally have only have one pre-existing condition.
- There is an increase in the number of pre-existing conditions as ge progresses.
- Patients who posses 3 pre-existing conditions, are found between the ages of 40 and 80.
- Female patients have a mean age of 38.89
- Male patients have a mean age of 33.74
- There a more male than female children under the age of 10. This is the only age bracket where the number of males is higher than females
- Male age distribution is more skewed to the right (skewness:0.25) than female age distribution(skewness: 0.09). This signifies that the difference between the mean and median is greater in males than in females.
Patients not enrolled in welfare attended more appointments than those enrolled in the welfare programs. Welfare does not have an impact on attendance.
- There are 10861 appointments of patients enrolled in the welfare program while the rest 99661 appointments are of patients not enrolled. Most appointments are of paients not enrolled in welfare.
- There are 5788 patients enrolled in the welfare.
- There are 56511 patients not enrolled in the welfare program.
- Patients enrolled in the welfare programs have less pre-existing conditions(Diabetes,Hipertension, Alcoholism, Handicap) when compared to patients who are not recipients of the program.
Saturday has the least number of appointments.
- There are no appointments on Sundays.
- The days ranked by total number of appointments in descending order are:
--Wednesday -- Tuesday -- Monday -- Friday -- Thursday --Saturday - Excluding Saturday, Thursday has the least number of missed appointments, followed by Friday. This may be due to their close proximity to the weekend.
- 7 am has the highest amount of scheduling occuring.
- People are least likely to schedule appointments during the night.
- There is less activity in terms of scheduling appointments as the day progresses.
- The trend in appointment attendance is the same in comparison to the total number of appointments for that day. This means Appointment Day does not affect attendance.
- Patients generally show up for appointments. This data requires specific time information for appointment days, to get a more accurate representation of attendance. Another question to be asked is, why scheduling of appointments decreases as the day progresses? The reason why Saturday has the least number of appointments regardless of being a weekend, requires further analysis.
Appointments with shorter number of waiting days have a higher attendance.
- The majority of waiting days are between 0 and 25 for attended appointments.
- The trend in distribution of waiting days is similar in both attended and missed appointments.
- The mean number of waiting days is 10 for all appointments.
- Attended appointments have a smaller mean of about 9 days.
- Missed appointments have a mean number of about 16 days.
LIMITATIONS
- Ilha de Trinidade is an island in the Pacific Ocean, and only 2 appoitments from this area are recorded in the dataset, and both show a missed attendance.The analysis cannot give an accurate representation of the patients in this neighbourhood.
- The appointment dates lack the time they were attended, this would assist in analysing attendance and determining which hours of the day or night are busiest in the hospital.
Resources¶
- Wikipedia : https://en.wikipedia.org/wiki/Trindade_and_Martim_Vaz Retrieved on 23-05-2022.
- Facts About Bolsa Família : https://thebrazilbusiness.com/article/facts-about-bolsa-familia Retrieved on 23-05-2022.
Submitting your Project¶
Tip: Before you submit your project, you need to create a .html or .pdf version of this notebook in the workspace here. To do that, run the code cell below. If it worked correctly, you should get a return code of 0, and you should see the generated .html file in the workspace directory (click on the orange Jupyter icon in the upper left).
Tip: Alternatively, you can download this report as .html via the File > Download as submenu, and then manually upload it into the workspace directory by clicking on the orange Jupyter icon in the upper left, then using the Upload button.
Tip: Once you've done this, you can submit your project by clicking on the "Submit Project" button in the lower right here. This will create and submit a zip file with this .ipynb doc and the .html or .pdf version you created. Congratulations!
In [124]:
from subprocess import call
call(['python', '-m', 'nbconvert', 'Investigate_a_Dataset.ipynb'])